Visual Spatial Reasoning

Abstract Spatial relations are a basic part of human cognition. However, they expressed in natural language variety ways, and previous work has suggested that current vision-and-language models (VLMs) struggle to capture relational information. In this paper, we present Visual Reasoning (VSR), dataset containing more than 10k text-image pairs with 66 types spatial English (e.g., under, front of, facing). While using seemingly simple annotation format, show how the includes challenging linguistic phenomena, such as varying reference frames. We demonstrate large gap between model performance: The ceiling is above 95%, while state-of-the-art only achieve around 70%. observe VLMs’ by-relation performances have little correlation number training examples tested general incapable recognising concerning orientations objects.1